The Parti - Game Algorithm

Parti-game is a new algorithm for learning feasible trajectories to goal regions in high dimensional continuous state-spaces. In high dimensions it is essential that learning does not plan uniformly over a state-space. Parti-game maintains a decision-tree partitioning of state-space and applies techniquesfrom game-theoryand computational geometryto eeciently and adaptively concentrate high resolution only on critical areas. The current version of the algorithm is designed to nd feasible paths or trajectories to goal regions in high dimensional spaces. Future versions will be designed to nd a solution that optimizes a real-valuedcriterion. Many simulated problems have been tested, ranging from two-dimensional to nine-dimensionalstate-spaces, including mazes, path planning, non-linear dynamics, and planar snake robots in restricted spaces. In all cases, a good solution is found in less than ten trials and a few minutes. is a promising method for robots to program and improve themselves. This paper addresses one of reinforcement learning's biggest stumbling blocks: the curse of dimensionality 5], in which costs increase exponentially with the number of state variables. These costs include both the computational eeort required for planning and the physical amount of data that the control system must gather. The curse of dimensionality is also a problem in other areas of artiicial intelligence, such as planning and supervised learning. Much work has been performed with discrete state-spaces: in particular a class of Markov decision tasks known as grid worlds 29], 28]. Most potentially useful applications of reinforcement learning, however, take place in multidimensionalcon-tinuous state-spaces. The obvious way to transform such state-spaces into discrete problems involves quantizing them: partitioning the state-space into a multidimen-sional grid, and treating each box within the grid as an atomic object. Although this can be eeective (see, for instance, the pole balancing experiments of 19], 4]), the naive grid approach has a number of dangers which will be detailed in this paper.